Moving traffic-light detection system for an automated vehicle

ABSTRACT

A traffic-light-detection system that visually determines a light-state of a traffic-light proximate to an automated vehicle includes a camera, a controller, and optionally a radar. The camera and the radar are on a host-vehicle. The camera renders a series-of-images of a traffic-light proximate to a host-vehicle. The radar detects radar-returns from the traffic-light. The controller is configured to determine a motion-pattern of the traffic-light based on the series-of-images and/or the radar-returns, and select a preferred-image from the series-of-images based on the motion-pattern. The preferred-image shows a light-source of the traffic-light characterized as being most directed at the camera when the motion-pattern indicates that the traffic-light is moving. The controller is further configured to determine a light-state of the traffic-light based on the preferred-image.

TECHNICAL FIELD OF INVENTION

This disclosure generally relates to a traffic-light-detection system,and more particularly relates to a system that determines amotion-pattern of the traffic-light, and then selects a preferred-imagefrom a series-of-images provided by a camera based on themotion-pattern, where the preferred-image is an image that shows alight-source of the traffic-light characterized as being most directedat the camera.

BACKGROUND OF INVENTION

It has been observed that camera based vision systems used on automatedvehicles often have difficulty detecting the state of a traffic-signal(e.g. red, yellow, green) when the traffic-signal is moving due to, forexample, wind.

SUMMARY OF THE INVENTION

In accordance with one embodiment, a traffic-light-detection system thatvisually determines a light-state of a traffic-light proximate to anautomated vehicle is provided. The system includes a camera and acontroller. The camera is on a host-vehicle. The camera renders aseries-of-images of a traffic-light proximate to a host-vehicle. Thecontroller is in communication with the camera. The controller isconfigured to determine a motion-pattern of the traffic-light based onthe series-of-images, and then select a preferred-image from theseries-of-images. The preferred-image shows a light-source of thetraffic-light characterized as being most directed at the camera whenthe motion-pattern indicates that the traffic-light is moving. Thecontroller is further configured to determine a light-state of thetraffic-light based on the preferred-image.

In another embodiment, a traffic-light-detection system that visuallydetermines a light-state of a traffic-light proximate to an automatedvehicle is provided. The system includes a camera, a radar, and acontroller. The camera and the radar are on a host-vehicle. The camerarenders a series-of-images of a traffic-light proximate to ahost-vehicle. The radar detects radar-returns from the traffic-light.The controller is in communication with the camera and the radar. Thecontroller is configured to determine a motion-pattern of thetraffic-light based on the radar-returns, and select a preferred-imagefrom the series-of-images based on the motion-pattern. Thepreferred-image shows a light-source of the traffic-light characterizedas being most directed at the camera when the motion-pattern indicatesthat the traffic-light is moving. The controller is further configuredto determine a light-state of the traffic-light based on thepreferred-image.

Further features and advantages will appear more clearly on a reading ofthe following detailed description of the preferred embodiment, which isgiven by way of non-limiting example only and with reference to theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

The present invention will now be described, by way of example withreference to the accompanying drawings, in which:

FIG. 1 is a diagram of a traffic-light-detection system in accordancewith one embodiment;

FIG. 2 is a top view of an intersection encountered by the system ofFIG. 1 in accordance with one embodiment; and

FIGS. 3A and 3B are images from a series-of-images rendered by thesystem of FIG. 1 in accordance with one embodiment.

DETAILED DESCRIPTION

FIG. 1 illustrates a non-limiting example of a traffic-light-detectionsystem 10, hereafter referred to as the system 10. In general, thesystem 10 visually determines a light-state 18 (e.g. red, green yellow)of a traffic-light 20 proximate to, e.g. in front of, within fiftymeters (50 m) for example, an automated vehicle, e.g. a host-vehicle 12.Advantageously, the system 10 is able to determine the light-state 18when the traffic-light 20 is moving, e.g. swinging from a cable 40 (FIG.2) that spans an intersection 22. When the traffic-light 20 is moving,the light emitted by the traffic-light 20 appears to vary from theperspective of the host-vehicle 12 as the direction of the bore-site oflight emitted by the traffic-light 20 varies. The apparent variation inapparent-intensity and/or size of the illuminated light (e.g. red, greenyellow) on the traffic-light 20 may make it difficult for prior examplesof visual traffic light detection systems to determine the light-state18 of the traffic-light.

As used herein, the term automated vehicle may apply to instances whenthe host-vehicle 12 is being operated in an automated-mode 14, i.e. afully autonomous mode, where a human-operator (not shown) of thehost-vehicle 12 may do little more than designate a destination in orderto operate the host-vehicle 12. However, full automation is not arequirement. It is contemplated that the teachings presented herein areuseful when the host-vehicle 12 is operated in a manual-mode 16 wherethe degree or level of automation may be little more than providing anaudible or visual warning to the human-operator who is generally incontrol of the steering, accelerator, and brakes of the host-vehicle 12.For example, the system 10 may merely warn the human-operator as neededto, for example, avoid ‘running’ a red-light, i.e. traveling through anintersection 22 (FIG. 2) when the traffic-light 20 indicates that thehost-vehicle 12 should stop.

The system 10 includes a camera 24 on a host-vehicle 12. The camera 24may be a video-camera or a camera capable of taking periodically timedimages. Whatever type is used, the camera 24 needs to be capable torender a series-of-images 26 of the traffic-light 20. Those in the artwill recognize that there are a wide variety of commercially availablecameras that are suitable for this application. If the camera 24 is usedto determine a motion-pattern 34 of the traffic-light 20 if/when thetraffic-light 20 is moving, it is preferable that the camera 24 has oris characterized by a minimum frame-rate, ten frames-per-second (10 fps)for example. As used herein, the motion-pattern 34 may be acharacterization of the type of motion exhibited by the traffic-light20. For example, the motion-pattern 34 may be characterized as swingingforward-and-backward relative to the host-vehicle 12, side-to-side (i.e.sideways or left-and-right) relative to the host-vehicle 12, swingingdiagonally relative to the host-vehicle 12, oscillatory-rotating about avertical-axis of the traffic-light 20, oscillating vertically along thevertical-axis (bouncing), or any combination thereof. The details of howthe motion-pattern 34 is determined and how the series-of-images 26 areanalyzed will be explained in more detail later.

The system 10 optionally includes a radar 28 on the host-vehicle 12. Avariety of radar-devices are commercially available for automotiveapplications that would be suitable to emit a radar-signal toward thetraffic-light 20 and detect instances of radar-returns 30 reflected by,and/or returning from, the traffic-light 20. If the radar 28 is used todetermine a motion-pattern 34 of the traffic-light 20 when thetraffic-light 20 is moving, it is preferable that the radar 28 has or ischaracterized by a minimum frame-rate, ten frames-per-second (10 fps)for example.

The system 10 includes a controller 32 in communication with the camera24, and the radar 28 if the radar 28 is provided or included in theconfiguration of the system 10. The controller 32 may include aprocessor (not specifically shown) such as a microprocessor or othercontrol circuitry such as analog and/or digital control circuitryincluding an application specific integrated circuit (ASIC) forprocessing data as should be evident to those in the art. The controller32 may include memory (not specifically shown), including non-volatilememory, such as electrically erasable programmable read-only memory(EEPROM) for storing one or more routines, thresholds, and captureddata. The one or more routines may be executed by the processor toperform steps for determining the motion-pattern 34 and the light-state18 based on signals received by the controller 32 from the camera 24 andoptionally the radar 28 as described herein.

If the system 10 does not include the radar 28, then the controller 32may be configured to determine a motion-pattern 34 of the traffic-light20 based on the series-of-images 26, and then select a preferred-image36 from the series-of-images 26. If the traffic-light is not moving,then likely any or all of the images in the series-of-images 26 could beanalyzed to determine the light-state 18. Indeed, it is recognized thatbasing the determination of the light-state 18 on multiple imagesincreases the confidence-level of the determination of the light-state18. However, when the traffic-light 20 is moving, the position of theilluminated light (red, green, yellow) in each image of theseries-of-images 26 likely changes which can make it difficult to have ahigh confidence-level.

As used herein, the preferred-image 36 from the series-of-images 26 maybe characterized one or more of the images that shows, or was renderedwhen, a light-source 38 (FIGS. 3A and 3B) of the traffic-light 20 ischaracterized as being most directed at the camera 24 when/while themotion-pattern 34 indicates that the traffic-light 20 is moving. Thatis, the preferred-image 36 is selected from the series-of-images 26 asbeing the one or more images that is/are most likely to indicate thelight-state 18 with high-confidence. It is recognized that if themotion-pattern 34 is relatively periodic, i.e. relatively predictable,then there may be multiple instances of the preferred-image 36 presentin the series-of-images 26 that are temporally spaced apart inaccordance with the periodicity of motion or oscillation of thetraffic-light 20.

FIG. 2 illustrates a non-limiting example of an intersection 22 equippedwith a traffic-light 20 that is suspended above the intersection 22 by acable 40 that is attached to poles at each end of the cable 40. In thisnon-limiting example the traffic-light 20 is moving or swinging with amotion-pattern 34 that can be characterized as swinging diagonallyrelative to a view-perspective of the host-vehicle 12.

FIGS. 3A and 3B are non-limiting examples of images from theseries-of-images 26 that may have been taken by the camera 24 from theperspective illustrated in FIG. 2 while the traffic-light was moving inaccordance with the motion-pattern 34 illustrated in FIG. 2. FIG. 3A maybe characterized as an instance of the preferred-image 36 because thelight-source 38, the green light in this example, is well-directedtoward the camera 24, i.e. is pointed almost directly at the camera 24.By contrast, FIG. 3B may be characterized as an instance of adiscarded-image 44 from the series-of-images 26 because the light-sourceis not well-directed toward the camera 24, i.e. is pointed away from thecamera 24. If the motion-pattern 34 is relatively periodic, then theremay be multiple instances of the preferred-image 36 shown in FIG. 3Apresent in the series-of-images 26 that are temporally spaced apart andrepeat or reoccur on a periodic basis. It follows that the multipleinstances of the preferred-image 36 shown in FIG. 3A may be groupedtogether for image-processing to determine the light-state 18 based onmultiple instances of the preferred-image 36 and thereby determine thelight-state 18 with high-confidence.

As previously suggested, the system 10 may include the radar 28, so itfollows that the controller 32 would be in communication with the camera24 and the radar 28. In this radar-included embodiment of the system 10,the controller 32 may be configured to determine the motion-pattern 34of the traffic-light 20 based on the radar-returns 30, and then selectthe preferred-image 36 from the series-of-images 26 based on themotion-pattern 34 indicated by the radar 28 rather than what might beindicated by the camera 24. Alternatively, it is contemplated thatinformation from both the camera 24 and the radar 28 may be combined todetermine which images of the series-of-images 26 is the preferred-image36 or are the preferred-images. The radar-returns 30 may be analyzedusing a variety of techniques known to those in the radar arts todetermine the motion-pattern 34 of the traffic-light 20.

By way of example and not limitation, one technique may be to determine,based on the radar-returns 30, when a mid-point of a periodic motionoccurs, and then designate the image from the series-of-images 26 thattemporally coincides with the mid-point as the preferred-image 36. Inview of the motion-pattern 34 suggested in FIG. 2, the image thatcorresponds to the mid-point is likely that shown in FIG. 3A, whichcorresponds to the position of the traffic-light shown in FIG. 2. Thatis, it is believed that the mid-point is where the light-source 38 ofthe traffic-light 20 characterized as being most directed at the camera24. In contrast, an end-point of the motion-pattern 34, i.e. a point ofmaximum deflection away from the mid-point, may correspond to FIG. 3B.It is also contemplated that the series-of-images 26 would likelyinclude one or more images that is/are the opposite of FIG. 3B. That is,if FIG. 3B is characterized as the traffic-light 20 swinging toward andleftward relative to the camera 24, then there is expected to be anopposite image in the series-of-images that shows the traffic-light 20as swinging away-from and rightward relative to the camera 24.

Several non-limiting examples of how images of the traffic-light 20 canbe subjected to image-processing to help determine the motion-pattern 34and the light-state 18 will now be discussed. One option is to determinean apparent-size 46 of the light-source 38 in the series-of-images by,for example, counting the number of camera-pixels that detect thelight-source 38. It is recognized that counting the number ofcamera-pixels that detect the light-source 38 in each image of theseries-of-images 26 will include tracking the position of where thelight-source 38 appears in each of the images. Several algorithms fortracking a moving object in a series-of-images are known and could beused here. The preferred-image 36 is then selected based on which of theimages shows or renders the apparent-size 46 characterized as largest inthe series-of-images 26. That is, the preferred-image 36 is the imagethat has the greatest number of camera-pixels that detect thelight-source 38. In this example it may not be necessary to determinethe particular type of motion, e.g. forward-and-backward, side-to-side,diagonal, oscillatory-rotating, bouncing. The effect of variation of theapparent-size 46 due to swinging if the traffic-light 20 is evident byexamining FIGS. 3A and 3B.

Another option is for the controller 32 to be configured to determine anapparent-intensity 48 of the light-source 38 in the series-of-images 26.When the traffic-light 20 is characterized as being most directed at thecamera 24, it is expected that the apparent-intensity 48 of thelight-source will be the greatest. The preferred-image is then selectedfor having the apparent-intensity 48 characterized as greatest in imagesof the series-of-images 26. Here again, it may not be necessary todetermine the particular type of motion, e.g. forward-and-backward,side-to-side, diagonal, oscillatory-rotating, bouncing.

Another option is for the controller 32 to be configured to define abounding-box 50 about each image of the traffic-light 20 in theseries-of-images 26, and determine if the traffic-light 20 is movingand/or the motion-pattern 34 based on changes in a box-size of thebounding-box 50 over the series-of-images. As used herein, the box-sizemay consist of, or include, a box-height, a box-width, a box-area, orany combination thereof. For example, if the motion isforward-and-backward relative to the camera 24, the box-height wouldvary, but the box-width may be substantially unchanged. If themotion-pattern 34 is determine to be forward-and-backward, the selectionof the preferred-image 36 may be relatively critical to making areliable determination of the light-state 18 because of the substantialvariation in the apparent-intensity 48. In contrast, side-to-side motionmay cause greater variation in box-width and box-area when compared tovariation in box-height. However, since the apparent-intensity 48 maynot change substantially, the selection which image or images are thepreferred-image 36 may not be critical to making a reliabledetermination of the light-state 18.

It is contemplated that the controller 32 may be advantageously furtherconfigured to determine when the motion-pattern 34 is such that another-light 52 (FIG. 3B) may be periodically revealed to the camera 24.As shown in FIG. 3, the swinging of the traffic-light 20 causes thered-light that is not directed to the host-vehicle 12 to be revealed tothe camera 24. Alternatively, the other-light 52 could be a street-light(not shown) or advertisement-light (not shown) that is hidden behind thetraffic-signal in FIG. 3A. Knowledge that the motion-pattern 34 is suchthat the other-light 52 may be periodically revealed to the camera 24may help to prevent confusion about the light-state 18. For example, ifthe system 10 detects both the light-source 38 as a relatively constantintensity green-light and the other-light 52 as a flashing red-light,the vehicle-operation block 54 (FIG. 1) may erroneously elect to applythe brakes rather than continue through the intersection 22.

It is also contemplated that the controller 32 may be advantageouslyfurther configured to determine a motion-period 56 of the motion-pattern34, where the motion-period 56 may correspond to the period ofoscillation of the swinging-motion of the traffic-light 20. Given themotion-period 56, the controller 32 can predict which of future-images58 rendered by the camera 24 will be selected as the preferred-image 36.That is, rather than wait until a large number of images in theseries-of-images 26 have been rendered and analyzed to select thepreferred-image 36, the motion-period 56 can be used to predict when thenext instance of an image likely to be an instance of thepreferred-image 36 will occur. This will allow the system 10 to morequickly and confidently determine that the light-state 18 has recentlychanged.

The system 10 may also include, or have access to via wirelesscommunications, a digital-map 60 that indicates traffic-light-positionsof various instances of the traffic-light 20 that the host-vehicle 12may encounter.

Accordingly, a traffic-light-detection system (the system 10), acontroller 32 for the system 10, and a method of operating the system 10is provided. The system 10 provides for more reliable and quickerdetermination of the light-state 18 of a traffic-light 20 when thetraffic-light 20 is swinging or otherwise moving relative to ahost-vehicle 12 equipped with the system 10.

While this invention has been described in terms of the preferredembodiments thereof, it is not intended to be so limited, but ratheronly to the extent set forth in the claims that follow.

We claim:
 1. A traffic-light-detection system that visually determines alight-state of a traffic-light proximate to an automated vehicle, saidsystem comprising: a camera on a host-vehicle, said camera renders aseries-of-images of a traffic-light proximate to a host-vehicle; and acontroller in communication with the camera, wherein the controller isconfigured to determine a motion-pattern of the traffic-light based onthe series-of-images, select a preferred-image from theseries-of-images, wherein the preferred-image shows a light-source ofthe traffic-light characterized as being most directed at the camerawhen the motion-pattern indicates that the traffic-light is moving,determine a light-state of the traffic-light based on thepreferred-image.
 2. The system in accordance with claim 1, wherein thecontroller is further configured to determine an apparent-size of thelight-source in the series-of-images, and the preferred-image isselected for having the apparent-size characterized as largest in theseries-of-images.
 3. The system in accordance with claim 1, wherein thecontroller is further configured to determine an apparent-intensity ofthe light-source in the series-of-images, and the preferred-image isselected for having the apparent-intensity characterized as greatest inthe series-of-images.
 4. The system in accordance with claim 1, whereinthe controller is further configured to define a bounding-box about eachimage of the traffic-light in the series-of-images, and determine themotion-pattern based on changes in a box-size of the bounding-box overthe series-of-images.
 5. The system in accordance with claim 1, whereinthe controller is further configured to determine when themotion-pattern is such that an other-light is periodically revealed tothe camera.
 6. The system in accordance with claim 1, wherein thecontroller is further configured to determine a motion-period of themotion-pattern, and predict which of future-images rendered by thecamera will be selected as the preferred-image.
 7. The system inaccordance with claim 1, wherein the system includes a radar incommunication with the controller, and the controller is furtherconfigured to further determine the motion-pattern based onreturn-signals detected by the radar.
 8. A traffic-light-detectionsystem that visually determines a light-state of a traffic-lightproximate to an automated vehicle, said system comprising: a camera on ahost-vehicle, said camera renders a series-of-images of a traffic-lightproximate to a host-vehicle; a radar on the host-vehicle, said radardetects radar-returns from the traffic-light; and a controller incommunication with the camera and the radar, wherein the controller isconfigured to determine a motion-pattern of the traffic-light based onthe radar-returns, select a preferred-image from the series-of-imagesbased on the motion-pattern, wherein the preferred-image shows alight-source of the traffic-light characterized as being most directedat the camera when the motion-pattern indicates that the traffic-lightis moving, determine a light-state of the traffic-light based on thepreferred-image.
 9. The system in accordance with claim 8, wherein thecontroller is further configured to determine an apparent-size of thelight-source in the series-of-images, and the preferred-image isselected for having the apparent-size characterized as largest in theseries-of-images.
 10. The system in accordance with claim 8, wherein thecontroller is further configured to determine an apparent-intensity ofthe light-source in the series-of-images, and the preferred-image isselected for having the apparent-intensity characterized as greatest inthe series-of-images.
 11. The system in accordance with claim 8, whereinthe controller is further configured to define a bounding-box about eachimage of the traffic-light in the series-of-images, and furtherdetermine the motion-pattern based on changes in a box-size of thebounding-box over the series-of-images.
 12. The system in accordancewith claim 8, wherein the controller is further configured to determinewhen the motion-pattern is such that an other-light is periodicallyrevealed to the camera.
 13. The system in accordance with claim 8,wherein the controller is further configured to determine amotion-period of the motion-pattern, and predict which of future-imagesrendered by the camera will be selected as the preferred-image.